Zoom control device, imaging apparatus, control method of zoom control  device, and recording medium

ABSTRACT

A subject detection unit of an imaging apparatus detects a subject image from an image. An automatic zoom control unit determines whether the subject detection unit detects the subject image in a specific region and performs first control such that a zoom magnification is controlled on a telephoto side based on a determination result when the subject image is not detected in the specific region. The automatic zoom control unit changes the first control to second control when the subject detection unit detects the subject image in the specific region, and controls the zoom magnification based on a reference size and a sequentially detected size of the subject image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technology for detecting a subjectimage from an image and controlling zoom magnification.

2. Description of the Related Art

In the related art, there are imaging apparatuses that have a functionof driving a zoom lens and performing optical magnification (hereinafterreferred to as optical zoom) and a function of expanding a part of aphotographed image and performing electronic magnification (hereinafterreferred to as electronic zoom). The imaging apparatuses also have anautomatic zoom function (automatic magnification) of automaticallychanging a zoom magnification according to detection information ofsubjects. Hereinafter, an optical zoom operation and an electronic zoomoperation are collectively referred to as a zoom operation.

Japanese Patent Laid-Open No. 2012-95019 discloses the configuration ofa camera equipped with an automatic zoom function of constantlymaintaining the size of a subject. Japanese Patent Laid-Open No.2012-95019 discloses a method in which a photographer selects acomposition among a plurality of options, “face-up,” “bust-up,” and“whole body,” in an automatic zoom function. In automatic zoom controlof maintaining the size of a subject image, the size of the subjectimage serving as a basis of zoom control is decided based on a selectedcomposition. For example, when a photographer selects the composition“whole body,” a zoom operation is performed so that an image of thewhole body of a person that is a subject fits in a photographing screen.

Japanese Patent Laid-Open No. 2011-188065 discloses a system in which asubject is tracked by an electric camera platform so that the subjectfits at an angle of view and automatic photographing is performed by acamera installed on the electric camera platform. Japanese PatentLaid-Open No. 2011-188065 discloses a method of driving a zoom lens of acamera and searching for a subject when a subject is not detectable.

However, in Japanese Patent Laid-Open No. 2012-95019, when a subjectwhich is a photographing target is far away, there is a possibility ofthe subject being undetectable even when the subject is within a screenin the above-described automatic zoom function. In such a scene, theautomatic zoom function may not be performed based on subjectinformation. Accordingly, in order to perform an automatic zoom process,it is necessary for a user to drive a zoom lens through a manual zoommanipulation until a subject image has a size at which it can bedetected. However, convenience of the automatic zoom function ofperforming matching of an angle of view through a simple manipulationmay be accordingly impaired. In a device disclosed in Japanese PatentLaid-Open No. 2011-188065, a zoom lens is automatically driven to searchfor a subject when a subject is not detected through tracking of thesubject by the electric camera platform and the camera. However, thereis a possibility of the search for the subject being ending when asubject which is not a photographing target is detected.

SUMMARY OF THE INVENTION

The present invention provides automatic zoom control of automaticallysearching for a subject without erroneous detection of a subject otherthan a photographing target even when the subject is not detected.

According to an aspect of the present invention, there is provided azoom control device including: a subject detection unit configured todetect a subject image from an image; a determination unit configured todetermine whether the subject detection unit detects the subject imagein a specific region of the image; and a control unit configured toperform control of zoom magnification. In the control of the zoommagnification, the control unit performs first control such that thezoom magnification is controlled on a telephoto side when thedetermination unit determines that the subject image is not detected inthe specific region, and the control unit performs second control suchthat the zoom magnification is controlled based on a reference size anda size of the subject image sequentially detected by the subjectdetection unit when the determination unit determines that the subjectimage is detected in the specific region.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof a digital camera according to an embodiment of the present invention.

FIG. 2 is a diagram exemplifying a relation between a focal distance andthe position of a focus lens for each subject distance.

FIGS. 3 is an explanatory diagram illustrating a process of preventing asubject (object) from falling outside of a screen.

FIGS. 4A to 4C are explanatory diagrams illustrating a process ofpreventing a subject (person) from falling outside of a screen.

FIGS. 5A and 5B are explanatory diagrams illustrating a process ofpreventing a change in a size of a subject (person) inside a screen.

FIGS. 6A to 6E are diagrams for describing composition setting when asubject is a person.

FIG. 7 is a flowchart for describing the flow of a process for anautomatic zoom function.

FIG. 8 is a flowchart for describing a subject searching process.

FIGS. 9A and 9B are diagrams for describing a search end region of thesubject searching process.

FIGS. 10A and 10B are flowcharts for describing a subject designationprocess.

FIG. 11 is a flowchart for describing another example of the subjectdesignation process.

FIG. 12 is a flowchart for describing a reference size setting process.

FIG. 13 is a flowchart for describing a screen size calculation process.

FIG. 14 is a transition diagram for describing automatic compositiondetermination in composition setting “auto.”

FIGS. 15A to 15C are diagrams illustrating subject positions and screensizes in the outermost periphery.

FIG. 16 is a flowchart for describing a process of calculating thereference size in the composition setting “auto.”

FIG. 17 is a flowchart for describing a process of calculating a subjectposition ratio in a horizontal direction.

FIG. 18 is a flowchart for describing a process of calculating a subjectposition ratio in a vertical direction.

FIG. 19 is a table illustrating the number of faces for estimating thesubject position.

FIG. 20 is a flowchart for describing an automatic zoom control process.

FIG. 21 is a flowchart for describing a zoom operation.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described indetail according to the appended drawings. FIG. 1 is a block diagramillustrating an example of the configuration of a digital camera 100that has an automatic zoom function and is an example of an imagingdevice including a zoom control device according to the embodiment.

A lens group is held in a lens barrel 101. A zoom lens 102 is moved inan optical axis direction of the lens barrel 101 to adjust a focaldistance and optically change an angle of view (move a zoom position). Afocus lens 103 is moved in the optical axis direction of the lens barrel101 to perform focusing. A vibration-proof lens (image blur correctionlens) 104 corrects image blur caused due to camera shake or the like. Anaperture and shutter 105 that adjust an amount of light are used forexposure control. In the embodiment, the digital camera 100 is animaging apparatus in which the lens barrel 101 and a camera body areintegrated, but the invention is not limited thereto. The embodiment canalso be applied to an imaging system configured to include a camera bodyand an interchangeable lens detachably mounted on the camera body.

An image sensor 106 generates an imaging signal by receiving lightpassing through the lens barrel 101 and converting a subject image intoan electric signal through photoelectric conversion. The image sensor106 is, for example, a charge-coupled device (CCD) type or acomplementary metal oxide semiconductor (CMOS) type image sensor. Theimaging signal generated by the image sensor 106 is input to an imageprocessing circuit 107 and is subjected to various processes such as apixel interpolation process and a color conversion process. Image dataafter the various processes is stored in an image memory 108. The imagememory 108 is a storage device such as a dynamic random access memory(DRAM) or a static random access memory (SRAM).

The display unit 109 is configured to include a thin film transistor(TFT) drive type liquid crystal display (LCD) and displays aphotographed image (image data) or specific information (for example,photographed information). An electronic viewfinder (EVF) function bywhich a photographer performs alignment of an angle of view can beprovided through information display such as live view related to aphotographed image.

An aperture and shutter driving unit 110 calculates exposure controlvalues (an aperture value and a shutter speed) based on luminanceinformation obtained through image processing in the image processingcircuit 107 and drives the aperture and shutter 105 based on acalculation result. In this way, automatic exposure (AE) control isperformed. A vibration-proof lens driving unit 111 calculates adeflection amount applied to the digital camera 100 based on deflectiondetection information by an angular velocity sensor such as a gyrosensor. Depending on the calculation result, the vibration-proof lens104 is driven so that the deflection amount applied to the digitalcamera 100 is cancelled (reduced).

A focus lens driving unit 112 drives the focus lens 103. In theembodiment, the digital camera 100 performs automatic focusing (AF)control according to a contrast scheme. That is, based on focusinginformation (contrast evaluation value) of a photographing opticalsystem obtained through image processing in the image processing circuit107, the focus lens driving unit 112 drives the focus lens 103 so that asubject comes into focus. However, the invention is not limited thereto,but a phase difference AF scheme may be used as the AF control insteadof the contrast scheme. Further, the AF control may be performedaccording to a plurality of schemes such as a combination of thecontrast scheme and another scheme.

A zoom lens driving unit 113 drives the zoom lens 102 according to azoom manipulation instruction. A manipulation unit 117 includes a zoomlever or a zoom button as a zoom manipulation member used for thephotographer to instruct the camera to perform zooming. A system controlunit 114 performs control of detecting a manipulation amount and amanipulation direction of the zoom manipulation member used for a zoominstruction manipulation, calculating a zoom driving speed or a zoomdriving direction, and moving the zoom lens 102 in an optical axisaccording to a calculation result.

Image data generated through a photographing operation is transmitted toa recording unit 116 via an interface (I/F) unit 115 to be stored. Theimage data is recorded in one or both of an external recording mediumand a nonvolatile memory 118. The external recording medium is, forexample, a memory card that is mounted on the digital camera 100 foruse. The nonvolatile memory 118 is a storage medium embedded in thedigital camera 100. The memory 118 stores not only program data or imagedata but also setting information of the digital camera 100 orinformation regarding a zoom-in position or the like in an automaticzoom function to be described below.

A manipulation unit 117 includes a release switch for instructingphotographing start and an automatic zoom manipulation switch forinstructing start or end of the automatic zoom function in addition tothe foregoing zoom manipulation member. A signal from the manipulationunit 117 is transmitted to the system control unit 114.

The system control unit 114 includes a calculation device such as acentral processing unit (CPU). The system control unit 114 controls theentire digital camera 100 by transmitting a control command to each unitin response to a manipulation of the photographer. The system controlunit 114 executes various control programs stored in the memory 118, forexample, programs for performing control of the image sensor 106, AE/AFcontrol, zoom control (including an automatic zoom process), and thelike.

To maintain a focused state when an angle of view is changed throughoptical zoom, it is necessary to move the focus lens 103 to anappropriate focus position according to the position of the zoom lens102 when the lens barrel 101 is of a rear focus type. Such control isreferred to as computer zoom (CZ) control and is performed by a CZcontrol unit 119 in the system control unit 114. FIG. 2 is a diagramexemplifying a relation between a focal distance of the zoom lens and afocus position for each subject distance. In FIG. 2, the relationbetween the focal distance of the zoom lens and the focused focusposition is graphed as a data table indicated for each distance up to asubject. In the embodiment, the table is referred to as a focus camtable. In FIG. 2, the horizontal axis represents the focal distancecorresponding to the zoom position and the vertical axis represents thefocus position. A distance (subject distance) from the digital camera100 to a subject is exemplified on the side of each graph line.

The system control unit 114 performs a scan operation by controlling thefocus lens driving unit 112 and moving the focus lens 103 within apredetermined range when the AF control is performed. A focus positionwhich is a focused point is detected using a contrast evaluation valueor the like obtained during a scan operation according to a knownmethod. A subject distance is measured using the zoom position and thefocus position at that time with reference to the focus cam table.

Next, control related to the automatic zoom function in the systemcontrol unit 114 will be described. As illustrated in FIG. 1, the systemcontrol unit 114 includes the CZ control unit 119, an electronic zoomcontrol unit 120, an automatic zoom control unit (hereinafter referredto as an AZ control unit) 122, and a subject detection unit 123.

The digital camera 100 has an optical zoom function and an electroniczoom function, and the CZ control unit 119 and the zoom lens drivingunit 113 serve to control the optical zoom. The CZ control unit 119detects a zoom position of the zoom lens 102 for each predeterminedcontrol period at the time of a zoom operation. Then, the CZ controlunit 119 performs control of driving the focus lens 103 to track thefocus cam table at a subject distance measured in the AF controlaccording to the detected zoom position. In this way, it is possible toperform the optical zoom operation while maintaining the focused state.

On the other hand, the electronic zoom control unit 120 and the imagememory 108 serve to control the electronic zoom. The electronic zoomcontrol unit 120 realizes the electronic zoom function by clipping datain a target region from image data transmitted to the image memory 108.The electronic zoom control unit 120 realizes smooth electronic zoomdisplay by causing the display unit 109 to perform display whilegradually increasing a range clipped at a frame rate period of an imagecaptured in the image sensor 106.

The subject detection unit 123 detects a desired subject region fromimage data stored in the image memory 108. In the embodiment, a subjectdetection method (a face detection process or a color detection process)of detecting a subject (the face of a person or the like or an object)based on face information or color information included in the imagedata will be described.

The face detection process is a process of detecting a face regionpresent in image data by a known algorithm. For example, the subjectdetection unit 123 extracts a feature amount from a rectangular partialregion on the image data and compares the feature amount to a featureamount of a face prepared in advance. Then, the subject detection unit123 determines that the partial region is a face region when acorrelation value of both features exceeds a predetermined thresholdvalue. By repeating this determination process while changing acombination of a size, a disposition position, and a disposition angleof the partial region, it is possible to detect various face regionspresent in the image data.

In the color detection process, a process of storing color informationof a subject region designated according to a subject designation methodto be described below as a feature color is performed. The colordetection process is performed when a detection target subject is anobject (“object” other than a person). For example, an RGB signal or aluminance signal (Y signal) which are output signals from the imageprocessing circuit 107 and a color difference (R-Y or B-Y) signal areused as the color information. When a subject is detected, the subjectdetection unit 123 divides image data into a plurality of partialregions and calculates an average value of luminance and colordifferences for each partial region. The subject detection unit 123compares feature color information stored in advance to colorinformation of each region at the time of the detection of the subjectand sets a partial region in which a difference between the luminanceand the color difference is equal to or less than a predetermined amountas a candidate for a subject region. A process of setting a group ofpartial regions mutually adjacent in the candidate of the subject regionas a same-color region and setting a region in which the same-colorregion is within a predetermined size range as a final subject region isperformed.

The subject detection unit 123 calculates the size of the subject regionin the image data using subject distance information measured by the CZcontrol unit 119 and focal distance information of the zoom lens 102. Aposture detection unit 124 detects a posture (for example, normalposition/grip-up/grip-down) of the digital camera 100 based oninformation of an acceleration sensor. A shake detection unit 125detects a deflection state of the digital camera 100 based on angularvelocity information or the like by a gyro sensor by determination. Theshake detection unit 125 determines that the camera is handheld when adeflection amount (detected value) applied to the gyro sensor or thelike is equal to or greater than a predetermined amount (thresholdvalue), and determines that the camera is fixed on a tripod or the likewhen the deflection amount is less than the predetermined amount.Sensors of a detection unit that acquires control information of thevibration-proof lens driving unit 111 may also be configured to be usedas the acceleration sensor and the gyro sensor used to detect theposture and detect the shake.

Next, an overview of the automatic zoom function and the AZ control unit122 according to the embodiment will be described. In a camera notequipped with the automatic zoom function, the following manipulation isnecessary, for example, when a subject moves and falls outside of theframe while the photographer performs framing in a telephoto state andwaits to take a picture.

First, the photographer performs a zoom-out operation by manipulatingthe zoom manipulation member, and then searches for a subject. After thephotographer searches for the subject, the photographer performs a zoommanipulation again to adjust an angle of view until the angle of theview becomes a desired angle of view. For example, when the size of asubject image changes due to movement of the subject, the photographernecessarily manipulates the zoom manipulation member to adjust the sizeof the subject image.

Conversely, in a camera equipped with the automatic zoom function, thephotographer performs a manipulation of designating a subject on a touchpanel or the like to designate a subject desired to be photographedafter setting the automatic zoom function. When the automatic zoomfunction is set, a zoom operation is automatically performed so that thedesignated subject has a predetermined size near the center of an image.As the method of designating a subject, in addition to the touch panelmanipulation, for example, there is a method of designating a subjectnear the center of a screen when the photographer manipulates a specificbutton or a method of automatically selecting a main subject amongsubjects detected by a camera.

The subject detection unit 123 calculates the position or size of asubject region designated from the image memory 108 in image data. Amotion of a subject can be tracked by performing this process on sampledimage data continuously whenever an image is displayed as a live view.The AZ control unit 122 starts the zoom-out operation when an image of asubject being tracked is detected in a zoom-out region to be describedbelow or when an image of the detected subject becomes greater than apredetermined size. That is, the AZ control unit 122 instructs the CZcontrol unit 119 or the electronic zoom control unit 120 to performzoom-out in a wide angle direction. When the subject is detected in azoom-in region and the subject image fits within a range of apredetermined size, a zoom-in operation is performed to the telephotoside. Through such a process, the photographer may operate the camerawithout worrying about the zoom operation so that an image of a desiredsubject fits in a screen. Even when the subject falls outside of theframe, a zoom magnification is automatically changed. Therefore, it ispossible to perform alignment of an angle of view more simply.

Next, start conditions of a zoom-out operation or a zoom-in operationwill be described with reference to FIGS. 3 to 5B. FIG. 3 is explanatorydiagrams illustrating a process of preventing a subject (object) fromfalling outside of a screen. FIGS. 4A to 4C are explanatory diagramsillustrating a process of preventing a subject (person) from fallingoutside of a screen.

In FIGS. 3 to 4C, frames 300 a and 300 b are first tracking frames(hereinafter referred to as object tracking frames) for tracking asubject (object) and frames 400 a to 400 f are second tracking frames(hereinafter referred to as face tracking frames) for tracking a subject(the face of a person). Hereinafter, for descriptions that could applyto a person or an object, the object tracking frame and the facetracking frames are collectively referred to as a subject tracking framein some cases. The subject tracking frame is displayed to surround asubject image on an electronic viewfinder screen (EVF screen) of thedisplay unit 109 so that the photographer can see a designated subject.The position and size of the subject tracking frame on the screen iscalculated based on the face information and the color information bythe subject detection unit 123 and is updated at a frame rate period.

A process of preventing a subject (airplane) from falling outside of ascreen will be described with reference to FIG. 3. A zoom-out region ZO(hereinafter referred to as a ZO region) in a left figure of FIG. 3indicates a region more outside than a predetermined ratio in the entireangle of view (entire screen) displayed by the EVF. For example, thecentral point of the screen is 0% and the entire screen is 100%, and acase in which a position located at 80% of the entire screen is set as aboundary of the ZO region is assumed. In this case, a region of 80% to100% of the entire screen is the ZO region. When a part of the objecttracking frame 300 a enters the ZO region, the AZ control unit 122performs control such that a zoom-out operation starts. The AZ controlunit 122 stores a zoom magnification (corresponding to a zoom-in angleof view) before zoom movement in the memory 118. A target zoommagnification or a zoom speed during a zoom-out operation are set inadvance according to the size of a subject image or a movement speed.The target zoom magnification or the zoom speed may be appropriatelycalculated according to the size of a subject image or the movementspeed. The zoom-out operation is performed according to the target zoommagnification or the zoom speed. In this way, it is possible toefficiently prevent a subject from falling outside of the frame.

A right figure of FIG. 3 illustrates an angle of view when a zoom-outoperation corresponding to a predetermined zoom change amount isperformed from the angle of view in the left figure of FIG. 3. A zoom-inregion ZI (hereinafter referred to as a ZI region) illustrated in theright figure of FIG. 3 indicates a region more inside than apredetermined ratio at a zoom-in angle of view (an angle of view beforezoom-out) 301 in a subject search state. For example, the central pointof the screen is 0% and the zoom-in angle of view 301 is 100%, and acase in which a position located at 70% of the zoom-in angle of view 301is set as a boundary of the ZI region is assumed. In this case, a regionof 0% to 70% of the entire zoom-in angle of view 301 is the ZI region.At this time, for example, when a zoom-out magnification is ½, thezoom-in angle of view 301 has a size of 50% of the entire screen.Accordingly, the boundary of the ZI region is 70%×(½)=35% of the entirescreen. The ZI region can also be said to be a region of 0% to 35% ofthe entire screen. When the photographer changes a direction of thecamera so that the object tracking frame 300 b fits in the ZI region,the AZ control unit 122 performs control such that a zoom-in operationstarts.

Next, a process of preventing a subject (person) from falling outside ofa screen will be described with reference to FIGS. 4A to 4C. When asubject is a person and a part of the face tracking frame enters the ZOregion, a zoom-out operation starts. When the face tracking frame fitsin the ZI region, a zoom-in operation is performed. The case in which asubject is a person is different from the case in which a subject is anobject. Thus, since a movement direction of the subject can be predictedto some extent, the ZO region and the ZI region are set according to aregion in the predicted movement direction. In photographing in thehandheld state, a subject may fall outside of the frame due to aninfluence of camera shake or the like. However, when the subject fallsoutside of the frame due to camera shake or the like, the photographercan include the subject in the frame again by performing an operation ofincluding the subject in the frame. Here, when a ZO region is set in anupper portion of the screen and a person is disposed near the center toperform photographing, the face tracking frame enters the ZO region andthe subject may fall outside of the frame unintentionally. Accordingly,when a subject is a person in the handheld state (a state in which thephotographer holds the camera), no ZO region is set in an upper portionof the screen in consideration of a framing operation of thephotographer.

In this way, in the embodiment, when the subject detection unit 123detects a face, the AZ control unit 122 changes the ZO region and the ZIregion according to a posture of the camera or a shake state. Theposture of the camera is detected by the posture detection unit 124 andthe shake state is determined from a detection result obtained by theshake detection unit 125. The detection result obtained by the shakedetection unit 125 refers to a detection result indicating whether thecamera is in the handheld state. Hereinafter, this will be describedspecifically with reference to FIGS. 4A to 4C.

FIG. 4A illustrates a ZO region and a ZI region set when thephotographer holds the camera and poses at a normal position. In such aphotographing scene, when a subject moves in a horizontal direction andfalls outside of the frame, the position of the subject in a screenmoves in the horizontal direction (longitudinal direction) of the screenat the normal position. Thus, both of the ZO region and the ZI regionare arranged in a vertical belt-like shape in the vertical direction(transverse direction) with respect to the screen at the normalposition. The ZO region is set in the vertical belt-like shape locatedon both end sides in the horizontal direction in a horizontally longrectangular screen. The ZI region is set in the vertical belt-like shapelocated in the central portion of the screen. In this case, when theface tracking frame 400 a enters the ZO region, the AZ control unit 122determines that zoom-out starts and performs control of the zoom-outoperation corresponding to a predetermined zoom magnification. When theface tracking frame 400 b is included in the ZI region, the AZ controlunit 122 determines that zoom-in starts and performs control of thezoom-in operation corresponding to a predetermined zoom magnification upto a zoom return position. By setting the ZO region and the ZI region inthis way, it is possible to efficiently prevent the subject (person)from falling outside of the frame.

FIG. 4B illustrates the ZO region and the ZI region set when the postureof the camera is changed in the same photographing scene and thephotographer poses the camera in a vertical position state of grip-downor grip-up. In this case, the ZO region and the ZI region are arrangedin vertical belt-like shapes in the vertical direction (longitudinaldirection) of a screen at a vertical position. That is, the ZO region isset in the vertical belt-like shape located on both end sides in thehorizontal direction in a vertically long rectangular screen. The ZIregion is set in the vertical belt-like shape located in the centralportion of the screen. In this case, when the face tracking frame 400 centers the ZO region, the AZ control unit 122 determines that zoom-outstarts and performs control of the zoom-out operation corresponding to apredetermined zoom magnification. When the face tracking frame 400 d isincluded in the ZI region, the AZ control unit 122 determines thatzoom-in starts and performs control of the zoom-in operationcorresponding to a predetermined zoom magnification up to a zoom returnposition. By setting the ZO region and the ZI region in this way, it ispossible to detect the motion of the subject in the horizontal directionand efficiently prevent the subject (person) from falling outside of theframe.

FIG. 4C illustrates the ZO region and the ZI region set when a detectionstate of the shake detection unit 125 is a fixed state. The fixed staterefers to a state in which the camera is fixed on a tripod or the like,and there is no possibility of a subject falling outside of the framedue to camera shake. When a zoom-in operation is performed while asubject is not framed near the center of a screen, there is apossibility of the subject falling outside of the frame due to thezoom-in operation. Accordingly, the ZO region is set in the entireperiphery portion of the screen and the ZI region is set further insidethan a zoom-in angle of view. That is, the ZO region is set in arectangular belt-like shape located on the corner sides in the verticaland horizontal directions of the screen. The ZI region is set in arectangular shape located in the center of the screen. In this case,when a face tracking frame 400 e enters the ZO region, the AZ controlunit 122 determines that zoom-out starts and performs control of thezoom-output operation corresponding to a predetermined zoommagnification. When a face tracking frame 400 f is included in the ZIregion, the AZ control unit 122 determines that zoom-in starts andperforms control of the zoom-in operation corresponding to apredetermined zoom magnification up to a zoom return position.

In the embodiment, as described above, the range of each of the ZOregion and the ZI region is dynamically changed according to a change inthe posture of the camera or the photographing state (handheldstate/fixed state). In this way, it is possible to efficiently prevent asubject from falling outside of a frame while preventing an erroneousoperation due to camera shake or the like. Further, the ZO region or theZI region may be changed according to one of the posture of the cameraand the photographing state (handheld state/fixed state) or only one ofthe ZO region and the ZI region may be changed.

Next, a zoom operation of maintaining a ratio of a subject image to ascreen within a predetermined range will be described. In theembodiment, control (size maintenance control) is performed so that thesize of the subject image is maintained within the predetermined rangefrom a reference size by automatically performing a zoom operation whenthe size of the detected subject image exceeds a predetermined multipleof the reference size. FIGS. 5A and 5B are diagrams for describing aprocess of maintaining the size of a subject (person) image in a screen.FIGS. 6A to 6E are explanatory diagrams of composition setting of asubject (person).

First, setting of a range (composition) in which a tracking targetsubject fits in a screen will be described with reference to FIGS. 6A to6E. FIG. 6A exemplifies screen display when composition setting issetting of “manual.” In the “manual” setting, the photographer changesthe size of a face tracked when the photographer performs a manual zoomoperation through a zoom lever manipulation while viewing an image of aperson on a screen. The size of the subject image at that time is storedas a reference size in the memory 118. FIG. 6B exemplifies screendisplay when the composition setting is setting of “face.” In thecomposition setting of “face,” a size with which a face fits in thescreen in accordance with a posture of the camera or a direction of theface is calculated as a reference size and is stored in the memory 118.Similarly, FIG. 6C exemplifies screen display when the compositionsetting is setting of “upper body” and FIG. 6D exemplifies screendisplay when the composition setting is setting of “whole body.” Areference size is calculated so that the reference size is a size set oneach screen and is stored in the memory 118.

FIG. 6E exemplifies screen display when the composition setting issetting of “auto.” In the “auto” setting, the AZ control unit 122determines an appropriate composition based on a subject position, asubject size, the number of subjects, a posture of the camera, or thelike in the screen. The reference size is calculated so that thedetermined composition is realized and is stored in the memory 118. Amethod of calculating the reference size will be described below withreference to FIG. 12.

The photographer can perform a manipulation of changing the compositionsetting by manipulating left and right buttons of the manipulation unit117 on a photographing screen or performing selection on a setting menu.When the photographer performs the manipulation of changing thecomposition setting, the AZ control unit 122 updates informationregarding the composition setting stored in the memory 118. FIGS. 6A to6E illustrate the composition setting examples of “manual,” “face,”“upper body,” and “auto” when the subject is a person, but the settingof the composition is not limited thereto. Only some of the fiveexemplified compositions may be set or other compositions may beincluded. When a subject is an object, for example, compositions of“manual,” “large,” “medium,” “small,” and “auto” may be set.

A case in which the composition setting is “manual” will be described asan example with reference to FIGS. 5A and 5B in the zoom operation ofmaintaining the ratio of the subject image to the screen within thepredetermined range. FIG. 5A exemplifies a zoom operation performedautomatically by the camera when a subject (person) approaches thecamera. This is a zoom-out operation of fitting the ratio of the subjectimage to the screen within the predetermined ratio. In FIGS. 5A and 5B,face tracking frames 500 a to 500 f are displayed to surround a faceregion as a feature region of the person who is the subject.Accordingly, the size of the face tracking frame will be described as asubject size herein.

A left figure of FIG. 5A illustrates an angle of view when a subject isdesignated according to a subject designation method to be describedbelow. The size of the face tracking frame 500 a at the time of thedesignation of the subject is stored as a reference subject size(reference size) in the memory 118. A center figure of FIG. 5Aillustrates an angle of view when the subject approaches the camera whenthe zoom magnification is not changed from the state of the left drawingin the left figure of FIG. 5A. For example, a size which is 150% of thesize of the face tracking frame 500 a which is the reference subjectsize is set as a start size of the zoom-out operation. When a relationbetween the subject tracking frames (face tracking frames) is “facetracking frame 500 b>face tracking frame 500 a×150%,” that is, thetracking frame is changed with respect to the reference size more than apredetermined change amount, the AZ control unit 122 determines that thezoom-out operation starts.

A right figure of FIG. 5A shows an angle of view zoomed out from theangle of view 501 in the center figure of FIG. 5A by a predeterminedzoom magnification and the face tracking frame 500 c. Here, thepredetermined zoom magnification is set to 1/1.5 in consideration of achange ratio (150%) of the size of the face tracking frame at the timeof the start of the zoom-out operation with respect to the referencesubject size. Thereafter, when the subject further approaches thecamera, the zoom-out to the wide angle side is further performed so thatthe subject image can continuously fit within the predetermined ratio.Accordingly, the photographer can concentrate on manipulating therelease switch.

In contrast, FIG. 5B exemplifies a zoom operation performedautomatically by the camera when the person who is a subject moves awayfrom the camera. This is a zoom-in operation of fitting a ratio of thesubject image to the screen within a predetermined range. A left figureof FIG. 5B shows an angle of view when a subject is designated accordingto a subject designation method to be described below. The size of theface tracking frame 500 d at the time of the designation of the subjectis stored as a reference subject size in the memory 118 (when thecomposition setting is “manual”).

A center figure of FIG. 5B shows an angle of view when the subject movesaway from the camera when the zoom magnification is not changed from thestate of the left figure of FIG. 5B. For example, a size which is 50% ofthe size of the face tracking frame 500 d which is the reference subjectsize is set as a start size of the zoom-in operation. When a relationbetween the face tracking frames is “face tracking frame 500 e>facetracking frame 500 d×50%,” the tracking frame is determined to bechanged with respect to the reference size more than a predeterminedchange amount. When the determination condition is satisfied and theface tracking frame 500 e is included in the ZI region, the AZ controlunit 122 determines that the zoom-in operation starts. Here, the ZIregion is set further inside than an angle of view 502 zoomed in by apredetermined zoom magnification with respect to the angle of view ofthe center figure of FIG. 5B.

A right figure of FIG. 5B shows an angle of view (corresponding to theangle of view 502) zoomed in from the angle of view in the center figureof FIG. 5B by the predetermined zoom magnification and the face trackingframe 500 f. Here, the predetermined zoom magnification is set to 1/0.5in consideration of a change ratio (50%) of the size of the facetracking frame at the time of the start of the zoom-in operation withrespect to the reference subject size.

The processes of preventing a subject from falling outside of a framehave been described when the subject is an object in FIG. 3 and when thesubject is a person in FIGS. 4A to 4C. The processes of fitting theratio of the size of the subject image to the screen within thepredetermined ranges have been described when the subject is a person inFIGS. 5A and 5B. Even when a tracking target subject is an object, zoomoperation start determination may be performed to control maintenance ofthe size of the subject image illustrated in FIGS. 5A and 5B as in thecase in which the subject is a person. When the composition setting is asetting other than “manual,” a zoom operation is automatically performedaccording to each reference size as in FIGS. 5A and 5B.

Next, a process for the automatic zoom function will be described withreference to FIGS. 7 to 21. FIG. 7 is a flowchart illustrating an entireprocessing example of the automatic zoom function. Hereinafter, theautomatic zoom function is assumed to be performed based on aninstruction of the system control unit 114 (the AZ control unit 122)unless explicitly mentioned otherwise.

In S700, the AZ control unit 122 first determines a manipulation stateof the automatic zoom manipulation switch of the manipulation unit 117.When the automatic zoom manipulation switch is pressed, the processproceeds to S701. When the automatic zoom manipulation switch is notpressed, the determination process of S700 is repeated. In S701, asubject searching process is performed. A subject designation process ofS702, a reference size setting process of S703, and automatic zoomcontrol of S704 are further performed in order.

Immediately after the process of the automatic zoom function starts orwhen a subject search button of the manipulation unit 117 is pressed(yes in S705), the subject searching process of S701 is performed. Thesubject searching process performed in step S701 in control of a zoommagnification will be described with reference to FIG. 8. When aphotographing target subject is far away from the camera, even though asubject image is in the screen, there is a possibility of the subjectbeing undetectable by the subject detection unit 123 when a featureregion of the subject is too small. In such a scene, the automatic zoomfunction based on subject information may not be performed. Accordingly,a subject searching process of performing a zoom-in operation whiledetecting a subject up to a predetermined focal distance is performed.

In S800, a process of determining whether the subject search button ispressed immediately after start of automatic zoom or during theautomatic zoom rather than immediately after the start of the automaticzoom is performed. In the case of the former positive determinationresult (yes) in S800, the process proceeds to S801. In the case of thenegative determination result (no), the process proceeds to S803. InS801, a process of acquiring a subject search end focal distance isperformed. The subject search end focal distance is a focal distance atan end zoom magnification when the searching of the subject image in theimage ends. Hereinafter, the subject search end focal distance isreferred to as a search end focal distance. In the embodiment, thesearch end focal distance is set to a focal distance of 85 mm by 35 mmsize conversion immediately after the start of the automatic zoom. Thefocal distance is a focal distance of a standard lens generallyconsidered to be appropriate when a person is photographed. Even whenthere is no subject (person) or a face is undetectable (for example, aperson is facing backward), the subject detection can be set to astandby state at an appropriate angle of view after the end of thesearching. Here, the focal distance is not limited to 85 mm. Anotherfocal distance (for example, 50 mm or 100 mm) corresponding to apredetermined (fixed) zoom magnification may be used irrespective of thezoom magnification at a current time. The photographer may change thesearch end focal distance from the menu setting or can change the searchend focal distance according to a photographing mode, a subject to bephotographed, or the like. When the process proceeds from S801 to S802,the focal distance at the time of mode start is compared to the searchend focal distance and it is determined whether the focal distance is ona wide angle side or a telephoto side from the search end focaldistance. When the focal distance at the time of the start of theautomatic zoom is on the telephoto side from the search end focaldistance (no in S802), the subject searching process ends. Conversely,when the focal distance at the time of the start of the automatic zoomis on the wide angle side from the search end focal distance (yes inS802), the process proceeds to S805.

Conversely, when it is determined in S800 that the subject is searchedfor through a manipulation of the subject search button, a process ofacquiring the search end focal distance is performed in S803. In theembodiment, the search end focal distance in the manipulation of thesubject search button is assumed to be a focal distance on the telephotoside 2.5 times the focal distance when the manipulation is performed.Here, this magnification is merely an example, and another magnificationmay be used. The photographer may change the search end focal distancefrom the menu setting or can change the search end focal distanceaccording to a photographing mode, a subject to be photographed, or thelike. The process proceeds from S803 to S804, the search end focaldistance acquired in S803 is compared to the focal distance at a teleend (telephoto end), and it is determined whether the search end focaldistance is on the wide angle side or the telephoto side from the focaldistance at the tele end. When the search end focal distance is on thetelephoto side from the focal distance at the tele end, the subjectsearching process ends. When the search end focal distance is on thewide angle side from the focal distance at the tele end, the processproceeds to S805. In S805, the AZ control unit 122 sets a zoom changeamount corresponding to the search end focal distance acquired in S801or S803 in the CZ control unit 119 or the electronic zoom control unit120 and instructs to perform the zoom-in operation.

Next, in S806, it is determined whether the current focal distancereaches the search end focal distance. When the current focal distancereaches the search end focal distance, the process proceeds to S809.After the zoom-in operation stops, the subject searching process ends.Conversely, when it is determined in S806 that the current focaldistance does not reach the search end focal distance, the processproceeds to S807. S807 is a process of determining whether a subject isdetected by the subject detection unit 123. When it is determined inS807 that the subject is detected, the process proceeds to S808. When itis determined that no subject is detected, the process returns to S806.In S808, it is determined whether a detection position of the subject iswithin a predetermined search end region of the screen. Thepredetermined search end region is assumed to be decided in accordancewith a threshold value (position determination reference value) set inadvance according to a subject. When the detection position of thesubject detected in S807 is within the predetermined search end regionon the screen, the process proceeds to S809, the zoom-in operationstops, and the subject searching process ends. Conversely, when it isdetermined in S808 that the detection position of the subject is notwithin the predetermined search end region on the screen, the processreturns to S806 and the determination processes from S806 to S808continue.

Here, the predetermined search end region described in S808 will bedescribed with reference to FIGS. 9A and 9B. When a specific subject (aperson or an object) is photographed, the subject which is aphotographing target is disposed near the center of the screen in somecases. When a subject which is not the photographing target is detectedin the periphery of the screen and the subject searching process ends,there is a possibility of the subject searching process ending beforethe subject which is the photographing target is detected. In this case,the automatic zoom function may not be performed according to thephotographer's intention. Accordingly, in the embodiment, only when asubject is detected in a central region of the screen which is aspecific region and within a region near this region, is the search endregion set as a region for determination so that the subject searchingprocess ends. In FIG. 9A, a search end region 900 a set when the subjectis an object (airplane) is exemplified. When the subject is an object, arange of a predetermined ratio, for example, 70%, from the center of thescreen in any of upward, downward, right, and left directions on thescreen is set as the search end region of the subject. In FIG. 9B, onthe other hand, a search end region 900 b set when the subject is aperson is exemplified. When the subject is a person, the search endregion is set so that a range of 100% from the center of the screen in aregion of an upper portion of the screen is included and a range of apredetermined ratio, for example, 70%, from the center of the screen inanother direction (right, left, or down) is included. When the subjectis a person, there is a possibility of a face tracking frame 902 of asubject which is a photographing target being disposed in the upperportion of the screen. For this reason, even when a subject is detectedin the upper portion of the screen, the search end region is set so thatthe subject searching process ends.

Next, the subject designation process indicated in step S702 in FIG. 7will be described with reference to FIGS. 10A and 10B. FIG. 10A is aflowchart illustrating an example of a process in which a userdesignates a desired subject using the touch panel included in themanipulation unit 117. In this example, the user is assumed to perform adesignating manipulation by touching a subject image displayed on thedisplay unit 109. In S1000, the AZ control unit 122 determines whetherthe touch panel is pressed. When the touch panel is pressed, the processproceeds to S1001. When the touch panel is not pressed, thedetermination process of S1000 is repeated.

In S1001, the AZ control unit 122 acquires information regarding aposition (touch position) touched by the user. Subsequently, in S1002,the AZ control unit 122 notifies the subject detection unit 123 of thetouch position and the subject detection unit 123 performs facedetection near the touch position. When the face is detected near thetouch position in S1002, a main subject is determined to be a person andthe process proceeds to S1003. Conversely, when the face is not detectednear the touch position in S1002, the main subject is determined to bean object other than a person and the process proceeds to S1004.

In S1003, the AZ control unit 122 performs control such that faceinformation of the person who is an automatic tracking target is storedin the memory 118. As the specific face information, there isinformation regarding a size of the face, a detection position of theface, a direction of the face, or the like when the subject isdesignated. In a camera having a face authentication function,identification information such as an authentication ID is also storedin the memory 118. In S1004, the AZ control unit 122 performs controlsuch that a feature color near the touch position is stored as colorinformation of the automatic tracking target in the memory 118. As thespecific color information, there is information regarding a featurecolor and its luminance, a chromatic value or a size of a same-colorregion, the central position of the same-color region, or the like atthe time of the subject designation. In the following description, theface information and the color information are collectively referred toas subject information (including a subject size and a subject detectionposition).

After the subject information is stored in the memory 118 in S1003 orS1004, the process proceeds to S1005 and the subject tracking frame isdisplayed. In S1005, the AZ control unit 122 performs control such thatthe subject detection position is set as a center and a subject trackingframe (an object tracking frame or a face tracking frame) with a sizecorresponding to the subject size is displayed on the display unit 109.Thereafter, the subject designation process ends.

In this way, the subject detection unit 123 in the system control unit114 detects a subject at a position designated on the display unit 109by the photographer or near the designated position. Then, the AZcontrol unit 122 causes the display unit 109 to display the subjecttracking frame. According to the example of the process of FIG. 10A, thesubject that the photographer desires to track can be designated simplyaccording to an intuitive method. However, the subject designationmethod is not limited to this method. Another method will be describedwith reference to FIG. 10B.

FIG. 10B is a flowchart illustrating an example of a process when theuser designates a subject using a different switch (subject designationswitch) from the automatic manipulation switch as a manipulation memberincluded in the manipulation unit 117. In S1006, the AZ control unit 122first displays a frame serving as a guide of the subject destinationnear the center of the screen of the display unit 109. The photographeradjusts the direction of the camera so that an image of a subject thathe or she desires to track using the frame as the guide is fitted nearthe center of the screen. Next, in S1007, the AZ control unit 122determines whether the subject designation switch is pressed. When thesubject designation switch is pressed, the process proceeds to S1008.When the subject destination switch is not pressed, the determinationprocess of S1007 is repeated in a standby state.

In S1008, the subject detection unit 123 performs the face detectionnear the center of the screen. When the face is detected near the centerof the screen, a main subject is determined to be a person and theprocess proceeds to S1009. Conversely, when the face is not detectednear the center of the screen, the main subject is determined to be anobject other than a person and the process proceeds to step S1010. Sincethe same process as S1003 of FIG. 10A is performed in S1009 and the sameprocess as S1004 of FIG. 10A is performed in S1010, the descriptionthereof will be omitted. After the subject information is stored in thememory 118 in S1009 or S1010, the process proceeds to S1011. Then, thesubject tracking frame (the object tracking frame or the face trackingframe) is displayed and the subject designation process ends.

In this way, in the processing example of FIG. 10B, the subjectdetection unit 123 detects the subject at the position of the center ofthe screen or near the position of the center of the screen of thedisplay unit 109. The AZ control unit 122 causes the display unit 109 todisplay the subject tracking frame indicating the position of thesubject. Even in a camera on which a manipulation member such as a touchpanel is not mounted, the user can simply designate a subject.

FIG. 11 is a flowchart illustrating an example of a process ofautomatically selecting a subject to be tracked among faces detectedwhen the automatic zoom manipulation switch included in the manipulationunit 117 is pressed. In S1100, the subject detection unit 123 firstperforms the face detection on the entire screen to determine whetherthe face of a person is detected. When the face of only one person isdetected on the entire screen, a main subject is determined to be aperson and the process proceeds to step S1101. Conversely, when one faceis not detected in the screen, the process proceeds to S1105.

In S1101, when the number of people whose faces are detected is oneperson, the subject detection unit 123 selects the face as a main face.When a plurality of faces are detected, the subject detection unit 123selects a main face assumed as a subject to be tracked among the faces.As the determination standard for selecting the main face, for example,there is a method of preferentially selecting a face of which a facedetection position is located closer to the vicinity of the center ofthe screen. When a plurality of faces are at substantially the sameposition, there is a method of selecting a face with a larger size as amain face. When a camera has a face authentication function, there is amethod of preferentially setting a face registered for authentication asa main face when the face registered for authentication is present.

The process proceeds to S1101 to S1102. Then, after the AZ control unit122 stores the face information regarding the main face in the memory118, the process proceeds to S1103 and the face tracking frame isdisplayed on the screen. Next, in S1104, the face changing determinationis performed. That is, when the plurality of faces are detected, theautomatically selected main face is not necessarily the face intended bythe photographer. For this reason, the photographer can change the mainface. At this time, when the photographer presses a switch (theautomatic zoom manipulation switch or the like) of the manipulation unit117 and designates a desired face, a process of changing the main faceto another face that was not selected as the main face among the facesfor which the face tracking frame was detected is performed. When themain face is changed in S1104, the process returns to S1102 and the faceinformation to be stored in the memory 118 is updated. In S1103, theface tracking frame is changed to the size and the detection position ofthe newly selected main face.

Conversely, the process proceeds from S1100 to S1105 when the mainsubject is determined to be an object other than a person. Then, the AZcontrol unit 122 performs control such that a feature color near thecenter of the screen is stored as color information of the automatictracking target in the memory 118. Next, in S1106, the subjectdesignation process ends after the object tracking frame is displayed onthe screen. The basic processes in S1102, S1105, S1103 (S1106) of FIG.11 are respectively the same as the processes in S1003, S1004, and 1005of FIG. 10A.

In this way, in the processing example of FIG. 11, the subject detectionunit 123 in the system control unit 114 performs the face detection onthe entire screen of the display unit 109. When a plurality of faces aredetected, the AZ control unit 122 performs control such that the displayunit 109 displays a subject tracking frame indicating the position of afirst face as a main subject among the plurality of faces. When thesubject is changed from the first face to a second face, the AZ controlunit 122 performs control such that the display unit 109 displays asubject tracking frame indicating the position of the second face. Thephotographer can designate the desired subject simply with fewermanipulations.

In the subject designation process, both of the method of FIG. 10A andthe method of FIG. 10B may be simultaneously applied when a cameraincludes both of the touch panel and the subject designation switchincluded in the manipulation unit 117. After the subject change isdetermined in S706 illustrated in FIG. 7, the subject can be designatedwith the method of FIG. 10A or 10B. After the subject searching processof S701, the subject can also be designated with the method of FIG. 11.The subject designation process applied according to a flow of theprocess may be changed. When the subject designation process (S702 ofFIG. 7) ends, the process proceeds to S703 of FIG. 7 to perform aprocess of setting the size of the tracked subject as a reference size.

The reference size setting process of S703 illustrated in FIG. 7 will bedescribed with reference to FIGS. 12 to 19. FIG. 12 is a flowchartillustrating an example of the reference size setting process when asubject is a person. In S1200, the AZ control unit 122 first determineswhether the composition setting stored in the memory 118 is “manual”described above. When the composition setting is “manual,” the processproceeds to S1201. When the composition setting is not “manual,” theprocess proceeds to S1203.

In S1201, the AZ control unit 122 performs subject detectiondetermination. When the subject detection unit 123 detects that thesubject is a person, the AZ control unit 122 determines whether the faceis detected. When the subject is an object, the AZ control unit 122determines whether the same feature color is detected. The determinationprocess of S1201 is repeated until the subject is detected. When thesubject is detected, the process proceeds to S1202. In S1202, the AZcontrol unit 122 sets the size of the detected subject image as areference size and stores the size in the memory 118. Then, the processends. Accordingly, when the composition setting is “manual,” the size(detection size) of the subject image first detected at the time ofstarting of the reference size setting process is stored as thereference size in the memory 118.

In S1203, the AZ control unit 122 calculates a screen size based on theposture information of the camera determined by the posture detectionunit 124. This screen size is used to calculate a reference size inS1211, S1704, S1705, S1806, and S1807 to be described below. A screensize calculation process will be described with reference to FIG. 13. Inthe embodiment, a case in which the screen size of a still image is setto “long side×short side=640×480” and the screen size of a moving imageis set to “long side×short side=640×360” will be exemplified. However,the screen size is not limited thereto.

In S1300, the posture detection unit 124 determines whether thephotographer holds the camera at the normal position or holds the cameraat the vertical position. When it is determined that the photographerholds the camera at the normal position, the process proceeds to S1301.When the photographer holds the camera at the vertical position, theprocess proceeds to S1306. In S1301, a process of setting 640 which is along side size to the screen size in the horizontal direction isperformed. Next, in S1302 a process of determining whether a current setmode is a mode in which a still image and a moving video can besimultaneously photographed (still image/moving image simultaneousphotographing mode) is performed. The still image/moving imagesimultaneous photographing mode is a mode in which the photographerpresses the release switch of the manipulation unit 117 duringphotographing of a moving image to simultaneously perform photographingof a still image or a form of a framing manipulation immediately beforephotographing of a still image is automatically recorded as a movingimage. When it is determined in S1302 that the current set mode is thestill image/moving image simultaneous photographing mode, a process ofcomparing the short side size of the moving image to the short side sizeof the still image and setting the smaller size as a vertical screensize is performed. This is because a subject is prevented from fallingoutside of a screen at any angle of view of the still image and themoving image by performing automatic zoom in accordance with the smallscreen size. Accordingly, in the embodiment, when the current set modeis the still image/moving image simultaneous photographing mode, theprocess proceeds to S1304 to set 360 which is a short side size of themoving image as the vertical screen size. Thereafter, the screen sizecalculation process ends.

When it is determined in S1302 that the current set mode is not theimage/moving image simultaneous photographing mode, the process proceedsto S1303. In S1303, it is determined whether a moving image is beingrecorded or a still image is being framed. When it is determined thatthe moving image is being recorded, the process proceeds to S1304.Conversely, when it is determined that the moving image is not beingrecorded, the process proceeds to S1305. In S1305, 480 which is a shortside size of the still image is set as the vertical screen size.

When it is determined in S1300 that the photographer holds the camera atthe vertical position, the process proceeds from S1306 to S1310. Theprocesses from S1306 to S1310 are processes in which the setting of thehorizontal screen size and the vertical screen size is switched in theprocesses from S1301 to S1305. That is, since “horizontal” and“vertical” can be replaced with each other in the above description, thedetailed description will be omitted. In the embodiment, the example inwhich the screen size of a still image is 4:3 (see FIGS. 15A and 15B)and the screen size of a moving image is 16:9 (see FIG. 15C) isdescribed, but the screen size may be changed according to an aspectratio. A ratio of an image of a person to a screen can be set to beconstant irrespective of the aspect ratio. The example in which thescreen size differs between the still image and the moving image in theimage/moving image simultaneous photographing mode has been described,but a plurality of still images or moving images with different screensizes may also be photographed simultaneously.

When the screen size is decided in S1203 of FIG. 12, the processproceeds to S1204. When the composition setting stored in the memory 118is one of “face,” “upper body,” and “whole body,” the AZ control unit122 decides a ratio (referred to as a face ratio) of the size of a faceimage to the screen according to the composition setting in S1204 toS1209. The face ratio according to the embodiment is a ratio of thelength of the face tracking frame to the screen in the verticaldirection, but may be a ratio of the length of the face tracking frameto the screen in the horizontal direction, an area ratio, or the like.Specifically, for example, it is determined in S1204 whether thecomposition setting is “face.” When the composition setting is “face”(yes in S1204), the face ratio is decided as 27% (S1207). When thecomposition setting is not “face,” the process proceeds to S1205 todetermine whether the composition setting is “upper body.” When thecomposition setting is “upper body” (yes in S1205), the face ratio isdecided as 16% (S1208). When the composition setting is not “upperbody,” the process proceeds to S1206 to determine whether thecomposition setting is “whole body.” When the composition setting is“whole body” (yes in S1206), the face ratio is decided as 7% (S1209).After the face ratio is decided in the steps from S1204 to S1209, theprocess proceeds to S1210.

In S1210, the AZ control unit 122 calculates the reference size bymultiplying the face ratio decided in S1204 to S1209 by the verticalscreen size calculated in S1203. This is a method of calculating thereference size from the face ratio to the screen size in the verticaldirection on the presupposition that the subject (person) is standing.When the direction of the face can be determined by the subjectdetection unit 123, the direction of the screen size may be decidedbased on the vertical direction of the face on the screen. Further,after the process of storing the calculated reference size in the memory118 is performed, the reference size setting process ends. Accordingly,when the composition setting is one of “face,” “upper body,” and “wholebody,” a size with a predetermined face ratio according to thecomposition setting to the screen size is set as the reference size.

When the composition setting is not “whole body” in S1206, that is, thecomposition setting is determined to be “auto,” the process proceeds toS1211. Then, the AZ control unit 122 automatically determines anappropriate composition according to a photographing state, the numberof subjects, a motion of the subject, or the like. An automaticcomposition determination and composition selection process in thecomposition setting of “auto” of S1211 will be described with referenceto FIG. 14. The composition setting “auto” is a mode in whichcomposition matching can be performed more simply by determining aphotographing scene and causing the camera to automatically select anappropriate composition (automatic composition selection).

FIG. 14 is a transition diagram of the composition automaticallyselected in the embodiment. In the embodiment, a composition selectionprocess of selecting an appropriate composition among five compositions,upper body 1400, whole body 1401, face 1402, multiple people 1403, andmanual 1404, according to a photographing scene is performed. In acomposition matching manipulation, it is necessary to set a subjectimage such that the subject image has an appropriate size instantlybefore photographing when a still image is photographed. On the otherhand, when a moving image is photographed, it is necessary to set asubject image such that the subject image normally has an appropriatesize during the photographing such as continuous tracking of a subjector approach of a stopped subject. When a moving image is photographed,image data at the time of a zoom-in operation or a zoom-out operationfrom a change in the composition is recorded. Therefore, a video can beeffectively photographed by changing the composition at an appropriatetiming. In this way, an appropriate composition differs between a stillimage and a moving image. Accordingly, a process of changing theselected composition in accordance with a photographing state isperformed during still image framing 1405 and during moving imagerecording 1406. That is, in the embodiment, a photographing preparationstate of a still image is exemplified as a first operation state and arecording state of a moving image is exemplified as a second operationstate. One composition is selected as the composition of a selectioncandidate belonging to a first composition group during the still imageframing 1405 from upper body 1400, whole body 1401, and multiple people1403. One composition is selected as the composition of a selectioncandidate belonging to a second composition group during the movingimage recording 1406 from upper body 1400, whole body 1401, and face1402. That is, the compositions of upper body 1400 and whole body 1401are common to the first and second composition groups, the compositionof multiple people 1403 is included in only the first composition group,and the composition of face 1402 is included in only the secondcomposition group. According to the specification of the imagingapparatus or the like, three or more operation states and a plurality ofcomposition groups (including a composition selectable in each operationstate) corresponding to the operation states can be set.

Here, determination conditions of photographing scenes and transition ofeach composition will be described. When the automatic compositiondetermination process S1211 of the composition setting “auto” starts,upper body 1400 is set as the composition of an initial state. Thecomposition of upper body 1400 is a composition selected when a stoppedsingular subject is determined to be a photographing scene during thestill image framing 1405 or during the moving image recording 1406. Insuch a scene, a composition of approach to a subject can beappropriately set by setting a composition in which the upper body ofthe subject is fitted in the screen.

When a motion of a photographing target subject is detected in thecomposition of upper body 1400 or when a distance between the camera andthe subject is determined to be greater than a predetermined distance D1(first threshold value), the composition is changed to whole body 1401.The composition of whole body 1401 is a composition selected when ascene in which a moving subject or a distant subject, that is, a subjectthat could easily fall outside of a screen, is photographed isdetermined during the still image framing 1405 or during the movingimage recording 1406. In such a scene, by tracking the subject in thecomposition in which the whole body of the subject is fitted, thesubject is set to fall outside of the screen as little as possible. Whenit is detected that the photographing target subject stops for apredetermined time or more in the composition of whole body 1401 and itis determined that the distance between the camera and the subject isless than a predetermined distance D2 (second threshold value), aprocess of returning the composition to upper body 1400 is performed.The predetermined time is a determination standard time set in advance.

Hereinafter, a method of determining a motion or stop of the subjectwill be described. When at least one of a change amount of position orsize of the subject detected by the subject detection unit 123 or ashake amount detected by the shake detection unit 125 in the detectionstate of the subject is greater than a predetermined amount, thephotographing target subject is determined to be moving. When the statein which both of the change amount of position or size of the subjectdetected by the subject detection unit 123 and the shake amount detectedby the shake detection unit 125 in the detection state of the subjectare less than the predetermined amount continues for a predeterminedtime or more, the photographing target subject is determined to bestopped. A time in which the detection state continues is assumed to bemeasured by a measurement timer.

Further, a method of calculating the distance between the camera and thesubject will be described. First, a focal distance serving as areference and a subject distance in regard to a subject size aremeasured in advance and measurement data is stored in the memory 118.The subject distance is calculated by performing calculation of “subjectdistance=(reference subject distance×focal distance×reference subjectsize)/(reference focal distance×detection size)” in regard to referencevalues stored in the memory 118. For example, when the reference valuesare the reference subject distance of 2 m, the reference focal distanceof 24 mm, and the reference subject size of 20 pix (pixels), a case inwhich a subject image with a detection size of 40 pix is detected with afocal distance of 120 mm is assumed. In this case, the subject distanceis calculated as “(2 m×120 mm×20 pix)/(24 mm×40 pix)=5 m.”

Next, a composition when multiple people are detected as subjects in ascreen will be described. When a plurality of people are detected assubjects in a screen in the composition of upper body 1400 or whole body1401 during the still image framing 1405, a process of changing thecomposition to multiple people 1403 is performed. The number of subjectsdetected at this time is stored in the memory 118. The composition ofmultiple people 1403 is a composition selected when all of the pluralityof subjects are determined to be a photographing scene of a group photofitted at a predetermined angle of view during the still image framing1405. In the composition state of multiple people 1403, a process ofdetermining whether the number of detected subjects is changed isperformed. When the number of detected subjects increases, a new subjectenters the screen, the number of subjects stored in the memory 118 isthus updated instantly, and the composition is changed. Conversely, whenthe number of detected subjects decreases, there is a possibility of atemporarily undetectable state, for example, as in a case in which onesubject is oriented horizontally. Therefore, when the composition isinstantly changed, there is a possibility of an undetectable subjecthaving fallen outside of the frame. Accordingly, when the number ofsubjects decreases, it is determined whether this state continues over apredetermined time or more. When this state continues for thepredetermined time or more, the number of subjects stored in the memory118 is updated and the composition is changed. Further, when the numberof subjects is one person, a process of returning the composition toupper body 1400 is performed.

When a composition in which all of the subjects are fitted at an angleof view is set during the moving image recording 1406 and a subjectother than the photographing target is moving, there is a concern of anerroneous operation of zoom occurring in response to this subject.Accordingly, when a plurality of people are detected as subjects duringthe moving image recording 1406, a process of setting only a mainsubject as a photographing target and maintaining upper body 1400 orwhole body 1401 as the composition is performed. When a face is detectednear the center of the screen over a predetermined time or more in thecomposition of upper body 1400 during the moving image recording 1406, aprocess of changing the composition to face 1402 is performed. Thecomposition of face 1402 is a composition selected when a scene in whichthe photographer is interested in the face of the subject is determinedduring the moving image recording 1406. In such a scene, the face of thesubject of interest can be photographed with a larger size by settingthe composition that approaches the face more than the upper body. Bychanging a zoom speed to a speed considerably lower or higher than anormal speed during a zoom-in operation by the change in thecomposition, a more effective video can be photographed. Further, when aface is detected near the periphery of a screen over a predeterminedtime or more in the composition of face 1402 or the photographer isdetermined to change framing, a process of returning the composition toupper body 1400 is performed. In the embodiment, a periphery region of ascreen is exemplified as a specific region, and a continuous time of astate in which the position of a face image of a subject detected by thesubject detection unit 123 is the periphery region is measured by ameasurement timer. As a method of determining the change of the framing,the photographer is determined to change the framing when the subjectdetection unit 123 does not detect the subject and the shake amountdetected by the shake detection unit 125 is greater than thepredetermined amount.

During the still image framing 1405, the photographer sometimescontinues to wait to take a picture in the same composition until thedirection or expression of the face of a person becomes a desired state.In this case, when the composition of approach to the face is set, thereis a possibility of the composition being a different composition fromthat intended by the photographer. Accordingly, even when the face isdetected near the center of the screen for a predetermined time or morein upper body 1400 during the still image framing 1405, a process ofmaintaining the composition of upper body 1400 is performed.

Next, a composition at the time of the change in a photographing stateby start/end of the moving image recording will be described. Thecompositions of upper body 1400 and whole body 1401 are compositionsselectable in both cases during the still image framing 1405 or duringthe moving image recording 1406. Accordingly, in these compositions, theoriginal composition is maintained even when the photographing state ischanged. On the other hand, the compositions of multiple people 1403 andface 1402 are compositions selected only in one case during the stillimage framing 1405 or during the moving image recording 1406.Accordingly, when the moving image recoding starts in the composition ofmultiple people 1403 during the still image framing 1405 or the movingimage recording ends in the composition of face 1402 during the movingimage recording 1406, the change to upper body 1400 which is a commoncomposition is considered. However, in this case, there is a concern ofan image being recorded in a zoom state in the beginning of a movingimage if the zoom operation starts along with the start or end of themoving image recording, or of the photographer feeling annoyed in thestill image framing. Accordingly, when the photographing state ischanged in such a state, a process of temporarily changing thecomposition to manual 1404 is performed. The composition of manual 1404is a composition in which the size of a main subject image firstdetected after the change of the composition is set to the referencesize. Accordingly, the zoom operation does not immediately start due tothe change of the photographing state and the photographer does not feelthe sense of discomfort. Even when a subject starts to move, the size ofa subject image can be maintained. Further, when a predetermined timehas passed in the composition of manual 1404, a process of returning thecomposition to upper body 1400 is performed.

As described above, when the composition setting is “auto” in S1211 ofFIG. 12, an appropriate composition is automatically determinedaccording to the photographing state, the number of subjects, a motionof the subject, or the like. Data indicating the composition after thedetermination is stored in the memory 118 and the process proceeds toS1212.

In S1212, it is determined whether the composition determined in S1211is changed from the immediately previous composition (whether thecomposition selected in FIG. 14 transitions or the number of subjects ischanged in the state of multiple people 1403). When the composition ischanged, the process proceeds to S1213 to determine whether the selectedcomposition is the composition of manual 1404. When the selectedcomposition is the composition of manual 1404, the process proceeds toS1201 to perform a process of storing the size of the first detectedsubject image as the reference size in the memory 118. When the selectedcomposition is not the composition of manual 1404, the process proceedsto S1215 to calculate the reference size in the composition setting“auto.” When it is determined in S1212 that the composition is notchanged, the process proceeds to S1214 to determine whether the subjectimage moves in the screen. As a method of calculating a movement amountof the subject image in the screen, a reference size is set in S1215,and subsequently the central position of the face tracking frameimmediately after the zoom operation up to the angle of view for thereference size is stored in the memory 118. A distance from the positionstored in the memory 118 to the central position of the detected facetracking frame is calculated as a movement amount. The central positionof the face tracking frame maybe added during each predetermined periodand the movement amount in the screen may be calculated. When thecalculated movement amount is equal to or greater than a predeterminedamount, the subject is determined to move. By normalizing thepredetermined amount with the size of the detected face tracking frame(setting a predetermined multiple of the size of the face trackingframe), a movement amount of an actual subject can be set to besubstantially the same irrespective of the size of the subject image inthe screen. When it is determined in S1214 that the subject moves in thescreen, the process proceeds to S1215 to calculate a reference size inthe composition setting “auto.” When it is determined in S1214 that thesubject does not move in the screen, the reference size setting processends without updating the reference size.

Hereinafter, the reference size calculation process in the compositionsetting “auto” will be described with reference to FIGS. 15A to 19. Whenthe composition setting is one of “face,” “upper body,” and “wholebody,” the reference size is calculated so that a face has a constantratio with respect to the screen size in S1210 of FIG. 12. That is, thesame reference size is calculated irrespective of the position of thesubject image on the screen. Therefore, when the composition is changedso that the subject image increases (is zoomed in on) in a case in whichthe subject is present in the periphery of the screen, there is apossibility of the subject falling outside of the frame. When thereference size is calculated with this method, it is necessary for thephotographer to frame the subject near the center of the screen.Accordingly, this method is not appropriate, for example, when thesubject is shifted from the center of the screen and is photographedalong with a background. When there are a plurality of subjects, it isnecessary to calculate the reference size with another method.Accordingly, when the composition setting is “auto,” the size of thesubject image is decided based on the composition determined in S1211and a process of calculating the reference size so that the subject isfitted in the screen irrespective of the position of the detectedsubject is performed.

FIGS. 15A to 15C are diagrams illustrating a relation between a screensize and the position of a subject closest to the periphery of thescreen among a plurality of subjects. FIGS. 15A and 15B illustratescreen examples of still image photographing and FIG. 15C illustrates ascreen example of moving image photographing and exemplifies a pluralityof people as subjects. FIG. 16 is a flowchart illustrating an overallprocess of calculating the reference size in the composition setting“auto.”

In S1600 of FIG. 16, a process of calculating a subject position ratioRh in the horizontal direction is first performed. The subject positionratio Rh in the horizontal direction is a ratio of a distance×2 up to ahorizontal position of a shoulder vicinity (hereinafter referred to as ashoulder position and denoted by Xsmax) of a subject closest to aperiphery from the center of a screen to a predetermined ratio withrespect to a screen size in the horizontal direction. The predeterminedratio is, for example, 80% or 90% of the screen size in the horizontaldirection and is changed according to the number of people arranged inthe horizontal direction in S1703 to be described below. In FIG. 15A, asubject closest to the periphery from the center of the screen in thehorizontal direction is a subject 1500 a among subjects 1500 a, 1501 a,and 1502 a. A process of calculating the subject position ratio Rh inthe horizontal direction will be described with reference to theflowchart of FIG. 17.

In step S1700, a shoulder position of the subject (denoted by Xs) iscalculated based on a central position or size of the face trackingframe of the subject. The center of the screen is assumed to be theorigin (0, 0), the central position of the face tracking frame isassumed to be (Xc, Yc), and the size of the face tracking frame isassumed to be S. When Ns is the number of face tracking frames locatedin a distance from the central position of the face tracking frame tothe shoulder position, the shoulder position Xs on the side of theperiphery of the screen can be calculated as “Xs=|Xc|+S×Ns”. FIG. 19illustrates a setting example of Ns. A value of Ns is changed accordingto the composition determined in S1211 of FIG. 12. For example, in thecomposition of face 1402, Ns=1 (in this case, a horizontal position ofthe vicinity of an ear rather than the shoulder position) is set. Inother cases (“upper body,” “whole body,” and “multiple people”), Ns=2 isset.

After the shoulder position Xs of the subject is calculated, the processproceeds to S1701. In S1701, the shoulder position Xs calculated inS1700 is stored as the maximum shoulder position Xsmax when the maximumvalue Xsmax of the shoulder position is not stored in the memory 118.When the maximum shoulder position Xsmax is stored in the memory 118,the shoulder position Xs calculated in S1700 is compared to the maximumshoulder position Xsmax. When the shoulder position Xs is greater thanthe maximum shoulder position Xsmax, a process of updating the maximumshoulder position Xsmax in the memory 118 to the shoulder position Xs isperformed. After the process of updating the maximum shoulder positionXsmax, the process proceeds to S1702. In S1702, it is determined whethera process of calculating the shoulder positions Xs of all the subjectsand updating the maximum shoulder positions Xsmax ends. When thecalculation of the shoulder position Xs and the updating of the maximumshoulder position Xsmax do not end, the process returns to S1700. Whenthe determination of the maximum shoulder positions Xsmax of all thesubjects ends and the updating of the maximum shoulder positions Xsmaxends, the process proceeds to S1703.

In S1703, a process of determining the number of people arranged assubjects in the horizontal direction (the number of people arranged inthe horizontal direction) is performed. This is because a composition inwhich the subjects are fully centered in a screen is set when the numberof subjects is larger as in a group photo, and a composition in which amargin remains in the periphery of the screen is set when the number ofsubjects is small. As a method of counting the number of people arrangedin the horizontal direction, when the face tracking frames overlap inthe vertical direction (the vertical direction of the screen), theoverlapping subjects are counted as one person in total. For example,when four people are detected as subjects in the screen and the facetracking frames of two people among the four people detected as subjectsoverlap in the vertical direction of the screen, the number of people iscounted as three. In S1703, the number of people arranged in thehorizontal direction is compared to a threshold value (for example, twopeople). When the number of people arranged in the horizontal directionis determined to be equal to or less than two, the process proceeds toS1704. When the number of people arranged in the horizontal direction isdetermined to be equal to or greater than three, the process proceeds toS1705. In S1704 and S1705, each subject position ratio Rh in thehorizontal direction is calculated. The subject position ratio Rh in thehorizontal direction in S1704 is calculated as a ratio of the maximumshoulder position Xsmax×2 to 80% of the horizontal screen size. Thesubject position ratio Rh in the horizontal direction in S1705 iscalculated as a ratio of the maximum shoulder position Xsmax×2 to 90% ofthe horizontal screen size. After S1704 or S1705, the process ofcalculating the subject position ratio Rh ends.

Next, a process of calculating the subject position ratio Rv in thevertical direction is performed in S1601 of FIG. 16. The subjectposition ratio Rv in the vertical direction is a ratio of a distance×2up to a vertical position of the head of the subject (hereinafterreferred to as a head position) or up to a vertical position of a body(hereinafter referred to as a body position) of the subject closest tothe periphery from the center of the screen to the predetermined ratioof the screen size in the vertical direction. The predetermined ratiois, for example, 90% of the screen size in the vertical direction.Hereinafter, the head position of the subject closest to the peripheryfrom the center of the screen is denoted by Yhmax and the body positionof the subject closest to the periphery from the center of the screen isdenoted by Ybmix. Of the subjects 1500 b, 1501 b, and 1502 b in FIG.15B, a subject whose head is closest to the periphery from the center ofthe screen in the vertical direction is the subject 1500 b, and asubject whose body is closest to the periphery from the center of thescreen is the subject 1501 b. A process of calculating the subjectposition ratio Rv in the vertical direction will be described withreference to the flowchart of FIG. 18.

In S1800 of FIG. 18, the head position (denoted by Yh) of the subject iscalculated based on the central position or size of the face trackingframe of the subject. The head position Yh can be calculated as“Yh=Yc+S×Nh”. Nh indicates the number of face tracking frames located ina distance from the central position of the face tracking frame to thehead position. A setting example of Nh is illustrated in FIG. 19. For avalue of Nh, Nh=1.5 is set irrespective of the composition determined inS1211 of FIG. 12. After the head position Yh of the subject iscalculated, the process proceeds to S1801.

In S1801, when a maximum value Yhmax of the head position is not storedin the memory 118, a process of storing the head position Yh calculatedin S1800 as a maximum head position Yhmax is performed. When the maximumhead position Yhmax is stored in the memory 118, the head position Yhcalculated in S1800 is compared to the maximum head position Yhmax. Whenthe head position Yh is greater than the maximum head position Yhmax, aprocess of updating the maximum head position Yhmax in the memory 118 tothe head position Yh is performed. After the updating process, theprocess proceeds to S1802.

Next, in S1802, a body position (denoted by Yb) of the subject iscalculated based on the central position or size of the face trackingframe of the subject. The body position Yb can be calculated as“Yb=Yc−S×Nb”. Nb indicates the number of face tracking frames located ina distance from the central position of the face tracking frame to thebody position. A setting example of Nb is illustrated in FIG. 19. Avalue of Nb is changed according the composition determined in S1211 ofFIG. 12. For example, in the case of the composition of face 1402,Nb=1.5 is set. In the case of the composition of upper body 1400, Nb=5is set. In the case of the composition of whole body 1401, Nb=10 is set.In the case of the composition of multiple people 1403, Nb=3.5 is set.In this setting, the center of the screen is set at a chest in face1402, at a part below the waist in upper body 1400, at a leg in wholebody 1401, and at a position near a part above the waist in multiplepeople 1403. After the body position Yb of the subject is calculated,the process proceeds to S1803.

In S1803, when a minimum value Ybmin of the body position is not storedin the memory 118, a process of storing the body position Yb calculatedin S1802 as a minimum body position Ybmin is performed. When the minimumbody position Ybmin is stored in the memory 118, the body position Ybcalculated in S1802 is compared to the minimum body position Ybmin. Whenthe body position Yb is less than the minimum body position Ybmin, aprocess of updating the minimum body position Ybmin in the memory 118 asthe body position Yb is performed. After the updating process, theprocess proceeds to S1804.

In S1804, the head positions Yh and the body positions Yb of all thesubjects are calculated and it is determined whether a process ofupdating the maximum head position Yhmax and the minimum body positionYbmin ends. When the updating of the maximum head position Yhmax and theminimum body position Ybmin do not end, the process returns to S1800.When the determination of the maximum head positions Yhmax and theminimum body positions Ybmin of all the subjects ends and the updatingof the maximum head positions Yhmax and the minimum body positions Ybminends, the process proceeds to S1805. In S1805, the absolute value of themaximum head position Yhmax is compared to the absolute value of theminimum body position Ybmin and a process of determining which value islocated closer to the periphery of the screen is performed. When it isdetermined that the maximum head position Yhmax is located closer to theperiphery of the screen, the process proceeds to S1806. When it isdetermined that the minimum body position Ybmin is located closer to theperiphery of the screen, the process proceeds to S1807. In S1806 andS1807, each subject position ratio Rv in the vertical direction iscalculated. In S1806, the subject position ratio Rv in the verticaldirection is calculated as a ratio of the absolute value×2 of themaximum head position Yhmax to 90% of the vertical screen size. InS1807, the subject position ratio Rv in the vertical direction iscalculated as a ratio of the absolute value×2 of the minimum bodyposition Ybmin to 90% of the vertical screen size. After S1806 or S1807,the process of calculating the subject position ratio Rv in the verticaldirection ends.

Subsequently, in S1602 of FIG. 16, the subject position ratio Rh in thehorizontal direction is compared to the subject position ratio Rv in thevertical direction. Accordingly, the position at which the ratio is thelargest among the ratios of the distances from the center of the screento the respective positions of the subject in regard to thepredetermined ratio of the screen size in each direction, that is, theposition closest to the periphery of the screen, can be determined. Therespective positions are the maximum shoulder position Xsmax, themaximum head position Yhmax, and the minimum body position Ybmin. Whenthe subject position ratio Rh in the horizontal direction is determinedto be greater than the subject position ratio Rv in the verticaldirection, the process proceeds to S1603. When the subject positionratio Rh in the horizontal direction is determined to be equal to orless than the subject position ratio Rv in the vertical direction, theprocess proceeds to S1604. In S1603 and S1604, a process of calculatingthe reference size is performed based on the detection size of the mainsubject and the ratio of the subject position closest to the peripheryof the screen determined in S1602. The reference size is calculated as“detection size/subject position ratio Rh in horizontal direction” inS1603 and is calculated as “detection size/subject position ratio Rv invertical direction” in S1604. That is, in S1603, the zoom operation isperformed so that the main subject image has the reference size obtainedby multiplying the size of the main subject image by “1/subject positionratio Rh in horizontal direction”. In S1604, the zoom operation isperformed so that the main subject image has the reference size obtainedby multiplying the size of the main subject image by “1/subject positionratio Rv in vertical direction”. In this way, the subject can be set atthe angle of view based on the composition determined in S1211 withoutthe subject falling outside of the screen.

When the composition setting is “auto” in S1215 of FIG. 12, the subjectposition closest to the screen periphery in the horizontal direction andthe vertical direction is determined based on the composition determinedin S1211 and the reference size is calculated so that this subjectposition is fitted in the screen. Accordingly, regardless of a positionon the screen at which the subject image is located, the subject can befitted at an appropriate angle of view without falling outside of theframe. Whether the number of subjects is singular or plural, thereference size can be calculated through the same process. In theembodiment, the direction of the screen size by which the reference sizeis calculated has been decided on the presupposition that the personserving as a subject is standing. The invention is not limited thereto.As long as the subject detection unit 123 can determine the direction ofthe face, the direction of the screen size may be decided based on thevertical direction of the face on the screen. Further, the compositiondetermination or the calculation example of the reference size when thesubject is a person has been described, but the invention can also beapplied when the subject is an object. However, in this case,compositions to be selected are assumed to be “large,” “medium,”“small,” and “multiple” instead of “face,” “upper body,” “whole body,”and “multiple people.” According to the compositions, the number ofobject tracking frames (corresponding to Ns, Nh, and Nb in the case of aperson) when the subject position is calculated may be set.

When the reference size setting process illustrated in FIG. 12 ends, theprocess proceeds to S704 of FIG. 7. In S704, the AZ control unit 122performs automatic zoom control based on the subject informationdetected by the subject detection unit 123 and the reference size storedin the memory 118. The automatic zoom control will be described belowwith reference to FIG. 20. After the automatic zoom control ends, theprocess proceeds to S705. Then, the AZ control unit 122 determineswhether the photographer gives an instruction to search for a subject.That is, the AZ control unit 122 determines whether the photographerpresses the subject search button of the manipulation unit 117. When thesubject search button is pressed, the process returns to S701 and thesubject searching process is performed. Conversely, when the subjectsearch button is not pressed, the process proceeds to S706. In S706, itis determined whether the automatic zoom target subject is changedthrough manipulation of the touch panel, a subject designation switch,or the like of the manipulation unit 117. When the subject is changed,the process returns to S702 to perform the subject designation process.Conversely, when the subject is not changed, the process proceeds toS707.

In S707, it is determined whether the composition setting is changedthrough manipulation of the left and right buttons of the manipulationunit 117 from the photographing screen. When the composition setting ischanged, data of the composition setting stored in the memory 118 isupdated, and subsequently the process returns to S703 to perform thereference size setting process. Conversely, when the composition settingis not changed, the process proceeds to S708. In S708, it is determinedwhether the automatic zoom manipulation switch of the manipulation unit117 is pressed. When the automatic zoom manipulation switch isdetermined to be pressed, the automatic zoom function ends. Conversely,when the automatic zoom manipulation switch is determined not to bepressed, the process proceeds to step S709. In S709, a process ofdetermining whether the composition setting stored in the memory 118 is“auto” is performed. When the composition setting is “auto,” the processreturns to S703 to perform the process of setting the reference sizeincluding the automatic composition determination. Conversely, when thecomposition setting is setting other than “auto,” the process returns toS704 to continue the automatic zoom control.

Next, the automatic zoom control process of S704 will be described withreference to the flowchart of FIG. 20. In S2000, the AZ control unit 122first determines whether the subject detection unit 123 detects thesubject. When the subject is not detected in S2000, the automatic zoomcontrol ends. When the subject is detected in S2000, the processproceeds to S2001. The determination of steps from S2001 to S2003 is anautomatic zoom start determination process for control of preventing asubject from falling outside of the frame. That is, this process isperformed to prevent a subject from falling outside of the screen, asdescribed in FIG. 3 to 4C. In S2001, the AZ control unit 122 determineswhether a subject tracking frame related to a tracking target subjectenters a ZO region. This ZO region corresponds to the ZO regiondescribed in the left figure of FIG. 3 and FIGS. 4A to 4C. When thesubject tracking frame enters the ZO region in S2001, that is, there isa high possibility of the subject falling outside of the frame, theprocess proceeds to S2004 to start a zoom-out operation. Here, thezoom-out operation corresponds to a zoom-out operation for the controlof preventing the subject from falling outside of the frame. After thezoom-out operation, the automatic zoom control ends.

Conversely, when the subject tracking frame does not enter the ZO regionin S2001, that is, when the camera captures the subject near the centerof the screen, the process proceeds to S2002. In S2002, it is determinedwhether an immediately previous zoom operation is a zoom-out operationperformed because the subject tracking frame entered the ZO region inS2001, that is, whether the immediately previous zoom operation is thezoom-out operation for the control of preventing the subject fromfalling outside of the frame. When it is determined in S2002 that theoperation is performed after the zoom-out operation for the control ofpreventing the subject from falling outside of the frame, the processproceeds to S2003. Conversely, when it is determined in S2002 that theoperation is not performed after the zoom-out operation for the controlof preventing the subject from falling outside of the frame, the processproceeds to S2006.

In S2003, the AZ control unit 122 determines whether the subjecttracking frame related to the tracking target subject is fitted(included) in a ZI region. Here, this ZI region corresponds to the ZIregion described in the right figure of FIG. 3B and FIGS. 4A to 4C. Whenit is determined in S2003 that the subject tracking frame is not fittedin the ZI region, the automatic zoom control process ends. Conversely,when it is determined that the subject tracking frame is fitted in theZI region, the process proceeds to S2005. That is, when the cameracaptures the subjects so that the subject size is within an angle ofview near the center of the screen and at a zoom return position, azoom-in operation starts in S2005. Here, the zoom-in operationcorresponds to the zoom-in operation for the control of preventing thesubject from falling outside of the frame. After the zoom-in operation,the automatic zoom control ends.

In the embodiment, to cause the control of preventing the subject fromfalling outside of the frame and the size maintenance control in theautomatic zoom to be compatible, the subject is captured near the centerof the screen for the control of preventing the subject from fallingoutside of the frame and the size maintenance control can be performed.Therefore, after the zoom-out operation for the control of preventingthe subject from falling outside of the frame, an automatic zoom process(S2006 to S2010 below) of constantly maintaining a subject size to bedescribed below (controlling size maintenance) is not performed. Inother words, when the control of preventing the subject from fallingoutside of the frame is performed, the size maintenance control is notperformed until the zoom-in operation for the control of preventing thesubject from falling outside of the frame is completed.

Next, processes of S2006 to S2010 will be described. When it isdetermined in S2002 that the operation is not performed after thezoom-out operation for the control of preventing the subject fromfalling outside of the frame, the process proceeds to S2006. The AZcontrol unit 122 compares the detection size of the subject to a sizeobtained by multiplying the reference size by a predetermined ratio(denoted by N1 where N1>1). The size of the subject indicated by theinformation regarding the reference subject set in S1202 or S1210 andS1215 of FIG. 12 is compared to the size of the subject detected inS2000. When the size of the subject detected in S2000 is greater than N1times the size of the reference subject, that is, a ratio of the subjectimage to the screen exceeds a predetermined value, the process proceedsto S2009. In S2009, the AZ control unit 122 starts a zoom-out operation.Here, this zoom-out operation corresponds to the zoom-out operation forthe size maintenance control. After the zoom-out operation, theautomatic zoom control ends.

Conversely, when the size of the subject detected in S2000 is equal toor less than N1 times the size of the reference subject in S2006, theprocess proceeds to S2007. In S2007, the AZ control unit 122 comparesthe size of the subject of the information regarding the referencesubject to the size of the subject detected in S2000. When the size ofthe detected subject is less than the size of the reference subject at apredetermined ratio (denoted by N2 where N2<1) (less than N2 times),that is, a ratio of the subject image to the screen is less than apredetermined value, the process proceeds to S2008. Conversely, when thesize of the detected subject is equal to or greater than N2 times thesize of the reference subject, the automatic zoom control ends.

In S2008, the AZ control unit 122 determines whether the subjecttracking frame related to the tracking target subject is fitted(included) in a ZI region. This is because the subject is prevented fromfalling outside of the frame by the zoom-in operation when the subjectis in the periphery of the screen. Here, this ZI region corresponds tothe ZI region described in the center figure of FIG. 5B. When it isdetermined in S2008 that the subject tracking frame is not fitted in theZI region, the automatic zoom control process ends.

Conversely, when it is determined in S2008 that the subject trackingframe is fitted in the ZI region, the process proceeds to S2010. InS2010, the AZ control unit 122 starts a zoom-in operation. Here, thiszoom-in operation corresponds to the zoom-in operation for the sizemaintenance control. In this way, in the embodiment, to prevent thesubject from falling outside of the frame even in the zoom-in operationfor the size maintenance control, the subject image is fitted in the ZIregion and then the zoom-in operation starts. After the zoom-inoperation, the automatic zoom control ends.

Next, a zoom operation will be described with reference to FIG. 21. FIG.21 is a flowchart for describing the zoom-out operation or the zoom-inoperation in S2004, S2005, S2009, and S2010 of FIG. 20. First, in S2100,the AZ control unit 122 acquires a zoom change amount (a change amountof zoom magnification) from the memory 118. In the case of the zoom-outoperation for the control of preventing the subject from falling outsideof the frame, a zoom-out change amount is set according to theinformation regarding the detected subject. Specifically, in thezoom-out operation for the control of preventing the subject fromfalling outside of the frame (S2004 of FIG. 20), the zoom-out changeamount is set to be smaller as the size of the subject is smaller.Accordingly, it is possible to prevent the subject from becoming toosmall due to the zoom-out operation and not being detected. Inconsideration of a detectable minimum size of the subject, the zoom-outoperation may not be performed when the size of the subject is less thana predetermined size. In a zoom-in operation for the control ofpreventing the subject from falling outside of the frame, a zoommagnification before start of the zoom-out operation is stored in thememory 118. A zoom-in change amount is set so that the zoommagnification is the same as the zoom magnification before the start ofthe zoom-out operation.

In the zoom-out operation for the size maintenance control (S2009 ofFIG. 20), a zoom-out change amount (1/N1 times) corresponding to apredetermined N1 times used for the determination of S2006 is set.Accordingly, even when the subject is undetectable, a minimum zoom-outoperation can be performed until the size of the subject becomes thesize of the reference subject. Similarly, in the zoom-in operation forthe size maintenance control (S2010 of FIG. 20), a zoom-in change amount(1/N2 times) corresponding to a predetermined N2 times used for thedetermination of S2007 is also set.

In S2101, the AZ control unit 122 sets the zoom change amount acquiredin S2100 in the CZ control unit 119 or the electronic zoom control unit120 and gives an instruction to perform a magnification process. Next,in S2102, the AZ control unit 122 determines which zoom operation isbeing performed between the zoom operations for the control ofpreventing the subject from falling outside of the frame and for thesize maintenance control. When the zoom operation at the current time isthe zoom operation for the control of preventing the subject fromfalling outside of the frame (S2004 and S2005 of FIG. 20), the processproceeds to S2105. Conversely, when the zoom operation at the currenttime is the zoom operation for the size maintenance control (S2009 andS2010 of FIG. 20), the process proceeds to S2103.

In S2103, the AZ control unit 122 determines whether the subjectdetection unit 123 detects the subject. When the subject is detected,the process proceeds to S2104. When the subject is not detected, theprocess proceeds to S2105. In S2104, the size of the subject indicatedby the information regarding the reference subject is compared to thesize of the subject detected in S2103. When the size of the subjectdetected in S2103 and the size of the reference subject are determinednot to be within a predetermined ratio range (within a predeterminedchange amount) as the result of the comparison, the process proceeds toS2102 again and the determination of the zoom operation continues. Whenthe size of the subject detected in S2103 and the size of the referencesubject are within the predetermined ratio range through the zoomoperation, the process proceeds to S2106. In S2106, the AZ control unit122 stops the zoom operation, and then ends the zoom operation process.

When it is determined in S2102 that the zoom operation is the zoomoperation for the control of preventing the subject from falling outsideof the frame or the subject is not detected during the zoom operationfor the size maintenance control in S2103, the process proceeds toS2105. In S2105, based on the zoom change amount acquired in S2100, theAZ control unit 122 determines whether a magnification processcorresponding to the zoom change amount of a predetermined amountaccording to each zoom operation is performed. When the magnificationprocess of the predetermined zoom change amount is not performed, theprocess returns to S2102 again to continue the process. Conversely, whenthe magnification process of the predetermined zoom change amount isperformed, the process proceeds to S2106 and the AZ control unit 122stops the zoom during the operation and ends the zoom operation process.

In the embodiment, as described above, the following control isperformed according to the determination result obtained according towhether the subject detection unit detects a subject image in a specificregion of an image:

-   -   first control: control in which a zoom position is set closer a        telephoto side than at a current time when the subject image is        not detected in the specific region; and    -   second control: control in which the zoom magnification is        controlled and a change in the size of the detected subject        image is suppressed with respect to the set reference size when        the subject image is detected in the specific region.

The specific region is set in the central region of an image, forexample, with respect to the main subject, but is changed according tothe type (people, objects, or the like), the number, the disposition, orthe like of the detected subject images.

In the embodiment, immediately after the automatic zoom function startsor when the subject search button is pressed, the process of searchingfor a subject is performed while the zoom-in operation up to the searchend focal distance is performed. Thereafter, when the subject image isdetected in the search end region set according to the subject, thezoom-in operation ends. Accordingly, even when no subject is detected,it is possible to perform the automatic zoom control of automaticallysearching for the subject without erroneous detection of a subject otherthan the photographing target.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiments and/or that includes one or morecircuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiments, and by a method performed by the computer of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described embodiments. The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD™, a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-260111, filed Dec. 24 2014, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. A zoom control device comprising: a subjectdetection unit configured to detect a subject image from an image; adetermination unit configured to determine whether the subject detectionunit detects the subject image in a specific region of the image; and acontrol unit configured to perform control of zoom magnification,wherein, in the control of the zoom magnification, the control unitperforms first control such that the zoom magnification is controlled ona telephoto side when the determination unit determines that the subjectimage is not detected in the specific region, and the control unitperforms second control such that the zoom magnification is controlledbased on a reference size and a size of the subject image sequentiallydetected by the subject detection unit when the determination unitdetermines that the subject image is detected in the specific region. 2.The zoom control device according to claim 1, wherein the specificregion includes a central region of the image.
 3. The zoom controldevice according to claim 1, wherein the specific region is set not toinclude at least a part of a periphery region of the image.
 4. The zoomcontrol device according to claim 1, wherein the determination unitdetermines the subject image detected by the subject detection unit andchanges the specific region.
 5. The zoom control device according toclaim 1, further comprising: a first instruction unit configured to givean instruction to start the control of the zoom magnification.
 6. Thezoom control device according to claim 5, further comprising: a secondinstruction unit configured to give an instruction to search for asubject during the second control, wherein the control unit starts thefirst control in accordance with the instruction of the secondinstruction unit.
 7. The zoom control device according to claim 1,wherein the control unit sets a first zoom magnification for determiningthe end of the first control and performs the first control until thezoom magnification reaches the first zoom magnification.
 8. The zoomcontrol device according to claim 7, wherein the control unit ends thefirst control when the determination unit determines that the subjectimage is not detected in the specific region and the zoom magnificationreaches the first zoom magnification.
 9. The zoom control deviceaccording to claim 7, wherein, when the control unit starts the firstcontrol according to an instruction of a first instruction unit givingan instruction to start the control of the zoom magnification, thecontrol unit sets a fixed zoom magnification as the first zoommagnification, irrespective of the zoom magnification before the startof the first control.
 10. The zoom control device according to claim 7,wherein, when the control unit starts the first control according to aninstruction of a second instruction unit giving an instruction to searchfor the subject image during the control of the zoom magnification, thecontrol unit sets the first zoom magnification to the zoom magnificationto be closer to the telephoto side more than the zoom magnificationbefore the start of the first control.
 11. The zoom control deviceaccording to claim 7, wherein, when the zoom magnification at a currenttime is closer to a telephoto side than the first zoom magnification,the control unit does not perform the first control, and when the zoommagnification at the current time is closer to a wide angle side thanthe first zoom magnification, the control unit performs the firstcontrol.
 12. The zoom control device according to claim 1, wherein thecontrol unit controls the zoom magnification in the second control sothat the size of the subject image sequentially detected by the subjectdetection unit is within a range of a predetermined change amount withrespect to the reference size.
 13. An imaging apparatus comprising: animaging unit configured to perform photoelectric conversion of a subjectimage to generate an image; a subject detection unit configured todetect the subject image from the image; a determination unit configuredto determine whether the subject detection unit detects the subjectimage in a specific region of the image; a zoom unit configured tochange a zoom magnification through lens driving or image processing;and a control unit configured to control the zoom unit, wherein thecontrol unit controls the zoom unit to perform first control such thatthe zoom magnification is controlled on a telephoto side when thedetermination unit determines that the subject image is not detected inthe specific region, and the control unit controls the zoom unit toperform second control such that the zoom magnification is controlledbased on a reference size and a size of the subject image sequentiallydetected by the subject detection unit when the determination unitdetermines that the subject image is detected in the specific region.14. A control method performed by a zoom control device, the methodcomprising: acquiring detection information of a subject image detectedfrom an image; determining whether the subject image is detected in thespecific region of the image based on the detection information acquiredin the acquiring; and performing first control such that the zoommagnification is controlled on a telephoto side when it is determinedthat the subject image is not detected in the specific region in thedetermining, and performing second control such that the zoommagnification is controlled based on a reference size and a size of thesubject image indicated by the detection information sequentiallyacquired when it is determined that the subject image is detected in thespecific region in the determining of whether the subject image isdetected.
 15. A non-transitory recording medium recording a controlprogram of a zoom control device which causes a computer to perform eachstep of a control method for the zoom control device, the methodcomprising: acquiring detection information of a subject image detectedfrom an image; determining whether the subject image is detected in thespecific region of the image based on the detection information acquiredin the acquiring; and performing first control such that the zoommagnification is controlled on a telephoto side when it is determinedthat the subject image is not detected in the specific region in thedetermining, and performing second control such that the zoommagnification is controlled based on a reference size and a size of thesubject image indicated by the detection information sequentiallyacquired when it is determined that the subject image is detected in thespecific region in the determining of whether the subject image isdetected.